{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WoegBf2gjiW4"
      },
      "source": [
        "> **NOTE**: This tutorial will demonstrate how to utilize the GPU provided by Colab to run LLM with Xinference local server, and how to interact with the model in different ways (OpenAI-Compatible endpoints/Xinference's builtin Client/LangChain).\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FAhwDgtUGIEo"
      },
      "source": [
        "# Xinference\n",
        "\n",
        "Xorbits Inference (Xinference) is an open-source platform to streamline the operation and integration of a wide array of AI models. With Xinference, you’re empowered to run inference using any open-source LLMs, embedding models, and multimodal models either in the cloud or on your own premises, and create robust AI-driven applications.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WzJegrOpGH4N"
      },
      "source": [
        "\n",
        "* [Docs](https://inference.readthedocs.io/en/latest/index.html)\n",
        "* [Built-in Models](https://inference.readthedocs.io/en/latest/models/builtin/index.html)\n",
        "* [Custom Models](https://inference.readthedocs.io/en/latest/models/custom.html)\n",
        "* [Deployment Docs](https://inference.readthedocs.io/en/latest/getting_started/using_xinference.html)\n",
        "* [Examples and Tutorials](https://inference.readthedocs.io/en/latest/examples/index.html)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LovckG0kGr9j"
      },
      "source": [
        "## Set up the environment\n",
        "\n",
        "> **NOTE**: We recommend you run this demo on a GPU. To change the runtime type: In the toolbar menu, click **Runtime** > **Change runtime typ**e > **Select the GPU (T4)**\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bPDeDltCGABt"
      },
      "source": [
        "### Check memory and GPU resources"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qhoItBBhF7uY",
        "outputId": "95957bb9-caa2-482b-b5bb-8224da56e20f"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "RAM: 12.67 GB\n",
            "=============GPU INFO=============\n",
            "Mon May 13 03:10:49 2024       \n",
            "+---------------------------------------------------------------------------------------+\n",
            "| NVIDIA-SMI 535.104.05             Driver Version: 535.104.05   CUDA Version: 12.2     |\n",
            "|-----------------------------------------+----------------------+----------------------+\n",
            "| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
            "| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |\n",
            "|                                         |                      |               MIG M. |\n",
            "|=========================================+======================+======================|\n",
            "|   0  Tesla T4                       Off | 00000000:00:04.0 Off |                    0 |\n",
            "| N/A   46C    P8               9W /  70W |      3MiB / 15360MiB |      0%      Default |\n",
            "|                                         |                      |                  N/A |\n",
            "+-----------------------------------------+----------------------+----------------------+\n",
            "                                                                                         \n",
            "+---------------------------------------------------------------------------------------+\n",
            "| Processes:                                                                            |\n",
            "|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |\n",
            "|        ID   ID                                                             Usage      |\n",
            "|=======================================================================================|\n",
            "|  No running processes found                                                           |\n",
            "+---------------------------------------------------------------------------------------+\n"
          ]
        }
      ],
      "source": [
        "import psutil\n",
        "import torch\n",
        "\n",
        "\n",
        "ram = psutil.virtual_memory()\n",
        "ram_total = ram.total / (1024**3)\n",
        "print('RAM: %.2f GB' % ram_total)\n",
        "\n",
        "print('=============GPU INFO=============')\n",
        "if torch.cuda.is_available():\n",
        "  !/opt/bin/nvidia-smi || ture\n",
        "else:\n",
        "  print('GPU NOT available')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eFzlnU4gG_JL"
      },
      "source": [
        "### Install Xinference and dependencies"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1eReutJA_jS_"
      },
      "outputs": [],
      "source": [
        "%pip install -U -q xinference[transformers] openai langchain"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vPq_TWiRQCAA",
        "outputId": "81d448f4-4c88-4e02-da25-206ce2d9b412"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Name: xinference\n",
            "Version: 0.11.0\n",
            "Summary: Model Serving Made Easy\n",
            "Home-page: https://github.com/xorbitsai/inference\n",
            "Author: Qin Xuye\n",
            "Author-email: qinxuye@xprobe.io\n",
            "License: Apache License 2.0\n",
            "Location: /usr/local/lib/python3.10/dist-packages\n",
            "Requires: aioprometheus, async-timeout, click, fastapi, fsspec, gradio, huggingface-hub, modelscope, openai, opencv-contrib-python, passlib, peft, pillow, pydantic, pynvml, python-jose, requests, s3fs, sse-starlette, tabulate, timm, torch, tqdm, typer, typing-extensions, uvicorn, xoscar\n",
            "Required-by: \n"
          ]
        }
      ],
      "source": [
        "!pip show xinference"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2EACA0GYHm2o"
      },
      "source": [
        "## A Quick Start Demo\n",
        "### Start Local Server\n",
        "\n",
        "\n",
        "To start a local instance of Xinference, run `xinference` in the background via `nohup`:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "5EM01Gq7IQ2y"
      },
      "outputs": [],
      "source": [
        "!nohup xinference-local  > xinference.log 2>&1 &"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WXtUJSC3I3kh"
      },
      "source": [
        "Congrats! You now have Xinference running in Colab machine. The default host and ip is 127.0.0.1 and 9997 respectively.\n",
        "\n",
        "\n",
        "Once Xinference is running, there are multiple ways we can try it: via the web UI, via cURL, via the command line, or via the Xinference’s python client."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0mkyrGIHJekz"
      },
      "source": [
        "The command line tool is `xinference`. You can list the commands that can be used by running:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "yayFuLIgJhYX",
        "outputId": "a12d7d9f-0c10-406e-9220-533bf8d8a315"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Usage: xinference [OPTIONS] COMMAND [ARGS]...\n",
            "\n",
            "  Xinference command-line interface for serving and deploying models.\n",
            "\n",
            "Options:\n",
            "  -v, --version       Show the current version of the Xinference tool.\n",
            "  --log-level TEXT    Set the logger level. Options listed from most log to\n",
            "                      least log are: DEBUG > INFO > WARNING > ERROR > CRITICAL\n",
            "                      (Default level is INFO)\n",
            "  -H, --host TEXT     Specify the host address for the Xinference server.\n",
            "  -p, --port INTEGER  Specify the port number for the Xinference server.\n",
            "  --help              Show this message and exit.\n",
            "\n",
            "Commands:\n",
            "  chat           Chat with a running LLM.\n",
            "  engine         Query the applicable inference engine by model name.\n",
            "  generate       Generate text using a running LLM.\n",
            "  launch         Launch a model with the Xinference framework with the...\n",
            "  list           List all running models in Xinference.\n",
            "  login          Login when the cluster is authenticated.\n",
            "  register       Register a new model with Xinference for deployment.\n",
            "  registrations  List all registered models in Xinference.\n",
            "  terminate      Terminate a deployed model through unique identifier...\n",
            "  unregister     Unregister a model from Xinference, removing it from...\n",
            "  vllm-models    Query and display models compatible with vLLM.\n"
          ]
        }
      ],
      "source": [
        "!xinference --help"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wvhIEjcHKXc5"
      },
      "source": [
        "### Run Qwen-Chat\n",
        "\n",
        "Xinference supports a variety of LLMs. Learn more in https://inference.readthedocs.io/en/latest/models/builtin/.\n",
        "\n",
        "Let’s start by running a built-in model: `Qwen-1_8B-Chat`.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z7OyMw8sKjj6"
      },
      "source": [
        "We can specify the model’s UID using the `--model-uid` or `-u` flag. If not specified, Xinference will generate it. This create a new model instance with unique ID `my-llvm`:\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "B_hQFqxOKiww",
        "outputId": "7c253c82-7a24-48df-c16a-a5da62f89106"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Launch model name: qwen-chat with kwargs: {}\n",
            "Model uid: my-llm\n"
          ]
        }
      ],
      "source": [
        "!xinference launch -u my-llm -n qwen-chat -s 1_8 -f pytorch -en transformers"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q--ic56eNDyo"
      },
      "source": [
        "When you start a model for the first time, Xinference will download the model parameters from HuggingFace, which might take a few minutes depending on the size of the model weights. We cache the model files locally, so there’s no need to redownload them for subsequent starts.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cfF-cCFlMCvE"
      },
      "source": [
        "## Interact with the running model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MYKNW0c-MONc"
      },
      "source": [
        "Congrats! You now have the model running by Xinference. Once the model is running, we can try it out either command line, via cURL, or via Xinference’s python client:\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VfZZham7Lj3X"
      },
      "source": [
        "### 1.Use the OpenAI compatible endpoint\n",
        "\n",
        "Xinference provides OpenAI-compatible APIs for its supported models, so you can use Xinference as a local drop-in replacement for OpenAI APIs. For example:\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GOStrwtRLehN",
        "outputId": "5b68b2b2-48bb-4e60-a5c7-8666e1fcab8a"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "ChatCompletion(id='chatff300498-10d7-11ef-97ae-0242ac1c000c', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content=\"As an AI language model, I don't have personal experiences or feelings, but I'm designed to assist and provide information on various topics. How can I help you today?\", role='assistant', function_call=None, tool_calls=None))], created=1715570535, model='my-llm', object='chat.completion', system_fingerprint=None, usage=CompletionUsage(completion_tokens=35, prompt_tokens=23, total_tokens=58))"
            ]
          },
          "metadata": {},
          "execution_count": 7
        }
      ],
      "source": [
        "import openai\n",
        "\n",
        "messages=[\n",
        "    {\n",
        "        \"role\": \"user\",\n",
        "        \"content\": \"Who are you?\"\n",
        "    }\n",
        "]\n",
        "\n",
        "client = openai.Client(api_key=\"empty\", base_url=f\"http://0.0.0.0:9997/v1\")\n",
        "client.chat.completions.create(\n",
        "    model=\"my-llm\",\n",
        "    messages=messages,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AmYB-_K4aXnG"
      },
      "source": [
        "### 2. Send request using curl\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "n7VLGirDaaR3",
        "outputId": "dc6d8001-4cdb-4e03-c3ad-4d02a7cb3737"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{\"id\":\"chat07fe7c94-10d8-11ef-88d5-0242ac1c000c\",\"object\":\"chat.completion\",\"created\":1715570550,\"model\":\"my-llm\",\"choices\":[{\"index\":0,\"message\":{\"role\":\"assistant\",\"content\":\"The largest animal in the world is the blue whale (Balaenoptera musculus). The blue whale can grow up to 100 feet long and weigh as much as 200 tons. It is the deepest-diving mammal in the world, capable of diving to depths of over 35,000 feet without using its breathing tube. Despite its massive size, the blue whale is relatively small compared to other marine animals, and it spends most of its life in the open ocean.\"},\"finish_reason\":\"stop\"}],\"usage\":{\"prompt_tokens\":25,\"completion_tokens\":105,\"total_tokens\":130}}"
          ]
        }
      ],
      "source": [
        "!curl -k -X 'POST' -N \\\n",
        "  'http://127.0.0.1:9997/v1/chat/completions' \\\n",
        "  -H 'accept: application/json' \\\n",
        "  -H 'Content-Type: application/json' \\\n",
        "  -d '{ \"model\": \"my-llm\", \"messages\": [ {\"role\": \"system\", \"content\": \"You are a helpful assistant.\" }, {\"role\": \"user\", \"content\": \"What is the largest animal?\"} ]}'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RJ_72F51XFZY"
      },
      "source": [
        "### 3. Use Xinference's Python client"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ohZPPubkXKLl",
        "outputId": "24d7832e-2683-444a-f6c7-7c906eeedd3b"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'id': 'chat1200b1f8-10d8-11ef-88d5-0242ac1c000c',\n",
              " 'object': 'chat.completion',\n",
              " 'created': 1715570567,\n",
              " 'model': 'my-llm',\n",
              " 'choices': [{'index': 0,\n",
              "   'message': {'role': 'assistant',\n",
              "    'content': 'Hello! How can I assist you today?'},\n",
              "   'finish_reason': 'stop'}],\n",
              " 'usage': {'prompt_tokens': 31, 'completion_tokens': 9, 'total_tokens': 40}}"
            ]
          },
          "metadata": {},
          "execution_count": 9
        }
      ],
      "source": [
        "from xinference.client import RESTfulClient\n",
        "client = RESTfulClient(\"http://127.0.0.1:9997\")\n",
        "model = client.get_model(\"my-llm\")\n",
        "model.chat(\n",
        "    prompt=\"hello\",\n",
        "    chat_history=[\n",
        "    {\n",
        "        \"role\": \"user\",\n",
        "        \"content\": \"What is the largest animal?\"\n",
        "    }]\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "P4PaU0fpdAuB"
      },
      "source": [
        "### 4. Langchain intergration"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ijeadB9DdDO8",
        "outputId": "4dcbed72-5e9d-442d-d0c8-ee5b86b375fd"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/langchain_core/_api/deprecation.py:119: LangChainDeprecationWarning: The class `LLMChain` was deprecated in LangChain 0.1.17 and will be removed in 0.3.0. Use RunnableSequence, e.g., `prompt | llm` instead.\n",
            "  warn_deprecated(\n",
            "/usr/local/lib/python3.10/dist-packages/langchain_core/_api/deprecation.py:119: LangChainDeprecationWarning: The method `Chain.run` was deprecated in langchain 0.1.0 and will be removed in 0.3.0. Use invoke instead.\n",
            "  warn_deprecated(\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "?\n",
            "The answer is:\n",
            "The tallest plant on Earth is a tree named Ginkgo biloba. It stands at a height of approximately 83 meters (276 feet) and has roots that reach deep into the ground. Ginkgo biloba is native to China but has been planted in gardens around the world as well. It is known for its beautiful, delicate leaves and its ability to survive even in harsh environments. Despite being over 100 years old, it continues to grow and thrive today.\n"
          ]
        }
      ],
      "source": [
        "from langchain.llms import Xinference\n",
        "from langchain.chains import LLMChain\n",
        "from langchain.prompts import PromptTemplate\n",
        "\n",
        "llm = Xinference(server_url='http://127.0.0.1:9997', model_uid='my-llm')\n",
        "\n",
        "template = 'What is the largest {kind} on the earth?'\n",
        "\n",
        "prompt = PromptTemplate(template=template, input_variables=['kind'])\n",
        "\n",
        "llm_chain = LLMChain(prompt=prompt, llm=llm)\n",
        "\n",
        "generated = llm_chain.run(kind='plant')\n",
        "print(generated)"
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "3NHxN3rF-XR2"
      },
      "execution_count": null,
      "outputs": []
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "gpuType": "T4",
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}